Method and Score Management Node For Supporting Evaluation of a Delivered Service

ABSTRACT

A method and score management node for supporting service evaluation of a service delivered by means of a telecommunication network. The score management node receives network measurements (v) related to at least one service event when the service is delivered to the user, and calculates a quality score Q for each received network measurement by applying a predefined scoring algorithm Q(v) on the network measurement. After identifying, among the calculated quality scores, quality scores Q which are related to a specific service session experienced by the user, the score management node determines a total session-specific quality score for the specific service session based on the identified quality scores Q. The determined total session-specific quality score can then be used for evaluation of the delivered service.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. provisional patentapplication No. 62/180,340, filed on Jun. 16, 2015, which isincorporated by reference.

TECHNICAL FIELD

The present disclosure relates generally to a method and a scoremanagement node for supporting service evaluation of a service deliveredby means of a telecommunication network.

BACKGROUND

When a service has been delivered by means of a telecommunicationnetwork by a service provider to one or more users, it is of interestfor the service provider to know whether the user is satisfied with thedelivered service or not, e.g. to find out if the service hasshortcomings that need to be improved in some way to make it moreattractive to this user and to other users. Service providers, e.g.network operators, are naturally interested in making their services asattractive as possible to users in order to increase sales, and aservice may therefore be designed and developed so as to meet the users'demands and expectations as far as possible. It is therefore useful togain knowledge about the users' opinion after service delivery in orderto evaluate the service. The services discussed in this disclosure may,without limitation, be related to streaming of audio and visual contente.g. music and video, on-line games, web browsing, file downloads, voiceand video calls, delivery of information e.g. in the form of files,images and notifications, and so forth, i.e. any service that can bedelivered by means of a telecommunication network.

A normal way to obtain the users' opinion about a delivered service isto explicitly ask the customer, after delivery, to answer certainquestions about the service in a survey or the like. For example, theservice provider may send out or otherwise present an inquiry form,questionnaire or opinion poll to the customer with various questionsrelated to user satisfaction of the service and its delivery. If severalusers respond to such a poll or questionnaire, the results can be usedfor evaluating the service, e.g. for finding improvements to make,provided that the responses are honest and that a significant number ofusers have answered. An example of using survey results for estimatingthe opinion of users is the so-called Net Promoter Score, NPS, which iscalculated from answers to user surveys to indicate the users' collectedopinions expressed in the survey answers.

However, it is often difficult to motivate a user to take the time andtrouble to actually answer the questions and send a response back to theservice provider. Users are often notoriously reluctant to provide theiropinions on such matters, particularly in view of the vast amounts ofinformation and questionnaires flooding users in the current modernsociety. One way to motivate the user is to reward him/her in some waywhen submitting a response, e.g. by giving some present or a discounteither on the purchased services or when buying future services, and soforth.

Even so, it is a problem that surveys can in practice only be conductedfor a limited number of users which may not be representative for allusers of a service, and that the feedback cannot be obtained in“real-time”, that is immediately after service delivery. A survey shouldnot be sent to a user too frequently either. The obtained feedback maythus get out-of-date.

Further problems include that considerable efforts and costs must bespent to distribute a survey to a significant but still limited numberof users and to review and evaluate all answers coming in, sometimeswith poor results due to low responsiveness. Furthermore, the user mayprovide opinions which are not really accurate or honest and responsesto surveys may even be misleading. For example, the user is often proneto forget how the service was actually perceived or experienced when itwas delivered, even after a short while, once prompted to respond to aquestionnaire. Human memory thus tends to change over time, and theresponse given may not necessarily reflect what the user really felt andthought at service delivery. The user may further provide the responsevery hastily and as simply as possible not caring much if it reallyreflects their true opinion. The opinion expressed may also be dependenton the user's current mood such that different opinions may be expressedat different occasions, making the response all the more erratic andunreliable.

Still another problem is that it can be quite difficult to trace anunderlying reason why users have been dissatisfied with a particularservice, so as to take actions to eliminate the fault and improve theservice and/or the network used for its delivery. Tracing the reason forsuch dissatisfaction may require that any negative opinions given byusers need to be correlated with certain operational specifics relatedto network performance, e.g. relating to where, when and how the servicewas delivered to these users. This kind of information is not generallyavailable and analysis of the network performance must be done manuallyby looking into usage history and history of network issues. Muchefforts and costs are thus required to enable tracing of such faults andshortcomings.

SUMMARY

It is an object of embodiments described herein to address at least someof the problems and issues outlined above. It is possible to achievethis object and others by using a method and a score management node asdefined in the attached independent claims.

According to one aspect, a method is performed by a score managementnode for supporting service evaluation of a service delivered to a userby means of a telecommunication network. In this method, the scoremanagement node receives network measurements related to at least oneservice event when the service is delivered to the user. The scoremanagement node further calculates a quality score Q for each receivednetwork measurement by applying a predefined scoring algorithm Q(v) onthe network measurement.

The score management then identifies, among the calculated qualityscores, quality scores Q which are related to a specific service sessionexperienced by the user, and determines a total session-specific qualityscore for the specific service session based on the identified qualityscores Q. The total session-specific quality score is used forevaluation of the delivered service.

According to another aspect, a score management node is arranged tosupport service evaluation of a service delivered to a user by means ofa telecommunication network. The score management node comprises aprocessor and a memory containing instructions executable by theprocessor, whereby the score management node is configured to operate asfollows.

The score management node is configured to receive network measurementsrelated to at least one service event when the service is delivered tothe user. The score management node is further configured to calculate aquality score Q for each received network measurement by applying apredefined scoring algorithm Q(v) on the network measurement, and toidentify, among the calculated quality scores, quality scores Q whichare related to a specific service session experienced by the user.

The score management node is also configured to determine a totalsession-specific quality score for the specific service session based onthe identified quality scores Q, wherein the total session-specificquality score is used for evaluation of the delivered service.

When employing the above method and/or score management node, thedetermined total session-specific quality score can be used in theservice evaluation as an estimation of the users' opinion of thespecific service session. Further, since the total session-specificquality score is calculated from technical measurements in the networkrelated to a specific service session, it is possible to evaluate theperformance of that service session based on the total session-specificquality score. The resulting total session-specific quality score canthus be regarded as a truthful estimation of the user's experience ofthe service when it was delivered in this specific service session.

A computer program storage product is also provided comprisinginstructions which, when executed on at least one processor in the scoremanagement node, cause the at least one processor to carry out themethod described above for the score management node.

The above method and score management node may be configured andimplemented according to different optional embodiments to accomplishfurther features and benefits, to be described below.

BRIEF DESCRIPTION OF DRAWINGS

The solution will now be described in more detail by means of exemplaryembodiments and with reference to the accompanying drawings, in which:

FIG. 1 is a block diagram illustrating an example of how a scoremanagement node may be configured and operate, according to somepossible embodiments.

FIG. 2 is a flow chart illustrating a procedure in a score managementnode, according to further possible embodiments.

FIG. 3 is a diagram illustrating an example of how a score managementnode may operate, according to further possible embodiments.

FIG. 4 is a diagram illustrating another example of how a scoremanagement node may operate, according to further possible embodiments.

FIG. 5 is a diagram illustrating another example of how a scoremanagement node may operate, according to further possible embodiments.

FIG. 6 is a diagram illustrating another example of how a scoremanagement node may operate, according to further possible embodiments.

FIG. 7 is a block diagram illustrating a score management node in moredetail, according to further possible embodiments.

DETAILED DESCRIPTION

The embodiments described in this disclosure can be used for supportingevaluation of a service by obtaining an estimated user opinion about theservice when it has been delivered to a user by means of atelecommunication network. The embodiments will be described in terms offunctionality in a “score management node”. Although the term scoremanagement node is used here, it could be substituted by “scoremanagement system” or similar term throughout this disclosure.

Briefly described, a session-specific quality score that reflects theuser's experience of the service, is determined based on technicalnetwork measurements made for at least one event or occasion when theservice was delivered to the user, hereafter referred to as a “serviceevent” for short. For example, the network measurements may relate tothe time needed to download data, the time from service request untildelivery, call drop rate, data rate and data error rate. The qualityscore is determined for a specific service session when the service isdelivered to the user which may generate several different measurementsin the network. The term “session-specific quality score” will be usedin this disclosure to denote a quality score that has thus beendetermined for a specific service session. The session-specific qualityscore thus reflects the user's experience of the service when it wasdelivered in this specific service session.

In the following description, any network measurements related todelivery of a service to the user by means of a telecommunicationnetwork are generally denoted “v” regardless of measurement type andmeasuring method. It is assumed that such network measurements v areavailable in the network, e.g. as provided from various sensors, probesand counters at different nodes in the network, which sensors, probesand counters are commonly used for other purposes in telecommunicationnetworks of today, thus being operative to provide the networkmeasurements v to the score management node for use in this solution.Key Performance Indicator, KPI, is a term often used in this field forparameters that in some way indicate network performance.

Further, the term “delivery of a service by means of a telecommunicationnetwork” may be interpreted broadly in the sense that it may also referto any service delivery that can be recorded in the network bymeasurements that somehow reflect the user's experience of the servicedelivery. Some further examples include services provided by operatorpersonal aided by an Operation and Support System, OSS, infrastructure.For example, “Point of sales” staff may be aided by various softwaretools for taking and executing orders from users. These tools may alsobe able to measure KPIs related to performance of the services. Anotherexample is the Customer Care personal in call centers who are aided bysome technical system that registers various user activities. Suchtechnical systems may as well make network measurements related to theseactivities as input to the score management node.

For example, the network measurements v may be sent regularly from thenetwork to the score management node, e.g. in a message using thehyper-text transfer protocol http or the file transfer protocol ftp overan IP (Internet Protocol) network. Otherwise the score management nodemay fetch the measurements v from a measurement storage where thenetwork stores the measurements. In this disclosure, the term “networkmeasurement v” may also refer to a KPI which is commonly prepared by thenetwork to reflect actual physical measurements. The concept of KPIs iswell-known as such in telecommunication networks. In this disclosure,the terms measurement, metric and KPI may be used interchangeably.

It will now be described how a score management node may operate, withreference to FIG. 1 and also to the flow chart in FIG. 2. FIG. 1illustrates a score management node 100 which receives networkmeasurements v made in a telecommunication network 102, while FIG. 2illustrates a procedure with actions performed by the score managementnode 100, to accomplish the functionality described in this disclosure.The score management node 100 is operative to support service evaluationof a service delivered to a user by means of a telecommunicationnetwork.

In this procedure, the network measurements v may be sent from thenetwork 102 more or less in real-time in a “live stream” fashion, e.g.from an Operation & Maintenance, O&M, node or similar, not shown.Alternatively, the network measurements v may be recorded by the networkin a suitable storage or database 104 which can be accessed by the scoremanagement node 100, e.g. at regular intervals.

The received network measurements v can be seen as “raw data” being usedas input in this procedure. For example, the above O&M node may be anaggregation point or node for distributed sensors and probes that makemeasurements in the traffic flows throughout the network. This node maycombine, correlate and generally process the measurement data in someway, e.g. to produce KPIs or the like.

A first action 200 illustrates that the score management node 100receives network measurements v related to at least one service eventwhen the service is delivered to the user. This operation may beperformed in different ways, e.g. when the network 102 sends a stream ofnetwork measurements as they are generated, or by fetching networkmeasurements from a measurement storage 104, as described above. Action200 may thus be executed continuously or regularly any time during thecourse of this process including the following actions. The protocolused in this communication may be the hyper-text transfer protocol httpor the file transfer protocol ftp, and the network measurements may bereceived in a message such as a regular http message or ftp message. Insome possible embodiments, the score management node may thus receivethe network measurements in a message according to the hyper-texttransfer protocol http or the file transfer protocol ftp.

In some further possible but non-limiting embodiments, the networkmeasurements may be related to any of: the time needed to download data,the time from service request until delivery, call drop rate, data rate,and data error rate. In another possible embodiment, the networkmeasurements may be made during a predefined time interval. FIG. 1illustrates that the network measurements are used to produce variousKPIs 106 which are obtained by the score management node 100.

In a next action 202, the score management node 100 calculates a qualityscore Q for each received network measurement by applying a predefinedscoring algorithm Q(v) on the network measurement. Some examples of howthe quality score Q may be calculated in this action by means of Q(v)will be described later below. A set of calculated quality scores Q isthen used as input in the next action 204 where the score managementnode identifies quality scores Q which are related to a specific servicesession experienced by the user. Actions 202 and 204 may be performed bya module for session scoring 108.

In a next action 206, the score management node 100 determines a totalsession-specific quality score for the service session based on theidentified quality scores Q. The total session-specific quality score isthen used for evaluation of the delivered service, which may be done indifferent possible ways. For example, different session-specific qualityscores may be determined for different service sessions when the servicewas delivered, and each session-specific quality score may be used as anestimation of a user's experience of the corresponding individualservice session or service event.

Another option is to use multiple session-specific quality scores as abasis for determining a more generic perception score P reflecting howthe service is perceived at any service session in general. A nextoptional action 208 illustrates that the score management node 100 may,in another possible embodiment, determine a perception score P based onthe total session-specific quality score that has been determined as ofactions 200-206, the perception score P reflecting a user experience ofa service, which may be performed by a module for perception scoring110. In this case, another possible embodiment is that the perceptionscore P may be determined based on several total session-specificquality scores calculated for multiple service sessions.

Finally, the perception score P may be made available, in an optionalaction 210, for use in evaluation of the service, as indicated bynumeral 116 in FIG. 1, e.g. by sending P to a service evaluation system118 or by saving P in a suitable storage, not shown in FIG. 1.

The protocol used for sending P to a service evaluation system 118 maybe e.g. the hyper-text transfer protocol http or the file transferprotocol ftp, and the perception score P may be sent to the serviceevaluation system 118 in an http message or an ftp message over an IPnetwork. The service evaluation system 118 or storage may comprise anSQL (Structured Query Language) database or any other suitable type ofdatabase.

In action 208, the perception score P may be calculated in severaldifferent ways and some illustrative but non-limiting examples of how itcould be calculated will be described in more detail later below. Itwill thus be described how the perception score P may be calculated fromnetwork measurements which is thus also applicable for session-specificquality scores.

Some further possible but non-limiting embodiments in the procedure ofFIG. 2 will now be described. In one possible embodiment, the totalsession-specific quality score may be determined based on a sessionscoring schema comprising a set of weighted measurement types such thatthe total session-specific quality score is calculated as a weightedaverage of the identified quality scores Q where each quality score Q isweighted according to the type of measurement used for determining saidquality score Q. Thereby, it may be selected how much influencedifferent measurement types will have on the resulting totalsession-specific quality score, which could be useful since a certainmeasurement type may be more relevant or significant to some servicesthan to other services. Similarly, a certain measurement type may bemore relevant or significant than other measurement types to aparticular service.

In another possible embodiment, the above-mentioned session scoringschema may be selected from a set of predefined session scoring schemasdepending on which types of network measurements have been made for theat least one service event. In another possible embodiment, each sessionscoring schema in the set of predefined session scoring schemas may inthis case comprise an entry criteria indicating at least one mandatorymeasurement type and/or at least one optional measurement type. Someexamples of how the above session scoring schemas, measurement types andentry criteria may be employed in this procedure, will be described inmore detail later below.

In another possible embodiment, the session scoring schema may beselected further depending on priorities assigned to the session scoringschemas in the set of predefined session scoring schemas. In anotherpossible embodiment, the session scoring schemas in the set ofpredefined session scoring schemas may be defined for different servicesor experience types.

In the above-described embodiments involving usage of different sessionscoring schemas and measurement types, the operation of determining thetotal session-specific quality score can be configured in a flexiblemanner and adapted to any kind of service delivery scenarios so as toachieve as accurate session-specific quality score as possible. Inanother possible embodiment, the measurement types discussed above mayinclude one or more Key Performance Indicators, KPIs.

Some examples of how the above-described scoring modules 108 and 110 maybe implemented in practice will now be outlined. Each of the scoringmodules 108 and 110 may be implemented as a piece of software executedby a suitable execution platform. This includes the possibility to havethe scoring modules 108 and 110 compiled into one program. In thisexample, the scoring modules may be software modules, e.g. in the formof Java classes, that are compiled into a single piece of software thatcontains the entire score calculation as exemplified above. A scoringcoordinator may be used for controlling the operation of each scoringmode.

Alternatively, a potentially more flexible implementation may be usedwhere the different operations described herein, e.g. the above scoringmodules 108 and 110, are treated as separate services implemented bydistinct pieces of software. These services could for example beService-Oriented Architecture, SOA, Web Services. It would also possibleto have the functions implemented as “worker nodes” in a streamprocessing environment such as “Storm”. In general, each functionalmodule may be a logical scoring node that can be realized in softwareand can be either co-deployed on one physical node or separated anddeployed into a set of physical processing nodes.

The scoring of user experience, referred to as perception scoring, maybe employed according to various practical implementations. Examples ofthis include Net Promoter Score (NPS), Mean Opinion Scores (MOS) andService Level Index (SLI). While the Net Promoter Score is merely basedon user studies, the MOS and SLI are used for obtaining predictivescores. These scores are mostly based on learned models that are used topredict user's opinion about his/her experiences for example when usingservices. Input to these models may typically be comprised of KPI andmetrics, i.e. the above-described network measurements, coming fromvarious sources in the network and the OSS/BSS infrastructure of theservice provider / operator. The MOS may be used as input to the ServiceLevel Index scoring.

Perception scoring based on metrics and KPI obtained from a networkand/or OSS/BSS infrastructure may have at least the following twoproblems:

(1) Missing Data: Network measurements may be missing due to temporaryerrors in the measurement and reporting infrastructure.

(2) Correlated Experience: Network measurements such as KPI and variousmetrics are collected broadly. For some service type there are morenetwork measurements available than for others. A simple scoring thatscores each measurement individually would involve many more data pointsfor experiences that happen to have more measurements. This can lead toan imbalance from higher representation of the underlying services. Thisunder- and overrepresentation of certain services is not based on actualinfluencing factors on user experience and perception, but it is theresult of an implementation detail.

Conventionally, each scoring, i.e. calculation of a perception score Preflecting a user experience of a service, is performed for eachindividual network measurement. In the solution described herein, it hasbeen realized that an improved scoring result may be obtained if thescoring would be done per service session rather than per individualmeasurement. To achieve such a session-specific perception or qualityscore, which may also be referred to as “session score” for short,measurements are first consolidated per service session to create asession score or quality score for each individual session, i.e. theabove-described session-specific quality score. An overall perceptionscore can then be calculated based on the session scores rather thandirectly based on the raw measurements.

MOS scores are already a kind of session scores which can be used asinput to perception scoring models, e.g. like the scoring model thatgenerates the SLI score. It is however a problem that MOS scores are notdefined for every service type and every user experience. Also the“maturity” of the underlying MOS calculation models may varyconsiderably.

Another problem that has been identified and realized in this solutionis that the notion of what a “service session” actually is may varyconsiderably depending on the type of service. A general definition inthe context of perception scores might be that any operation in thenetwork that results in a unique experience for a human user could beconsidered a “service session”. Sometimes this definition coincides witha technical session. An illustrative but non-limiting example of thiswould be a voice call.

Sometimes however, the technical definition of a service session doesnot help to understand the unique experience of the human user andtherefore the notion of a service session being used for perceptionscoring may be somewhat indefinite. Another typical example is mobileweb surfing. Every click made by the user leading to downloading awebsite would qualify to be a session in the technical sense. However,the human user would tend to consider all websites used until he/she hasfound the needed information, to be jointly a single unique experience.Here the technical notion of a service session is not very accurate orrelevant.

Another illustrative but non-limiting example may involve experiences inCustomer Care. The user may have called to report a problem. This singlephone call can be considered to be an “experience” for the user, but theentire process from reporting the error until it is solved can bedefined as a “customer care session” which could also be considered tobe a single unique experience for the user, even if it might involvemultiple technical sessions such as calls or the usage of otherchannels. In any case, it would be beneficial for the perception scoringif multiple network measurements could first be summarized into asession score.

In this disclosure, a method and apparatus for generating session scoresfrom any stream of input data, are described. It could be considered“opportunistic” in the sense, that it will make the best out of thepresented raw data. The session score, i.e. session-specific perceptionscore, generated according to embodiments described herein effectivelyexpresses the perceived quality of each experience. It is thereforegenerating something similar to a Mean Opinion Score.

In this procedure, a first operation may be to calculate a quality scorefor each of the individual network measurements. This produces a qualityscore Q for every measurement in the service session.

The second operation may then be to apply a configurable set of schemafor session score calculation, and this schema will be referred to as a“session scoring schema”. This schema can be used to calculate a totalsession-specific quality score for the service session from theindividual KPI specific quality scores.

It was mentioned above that some possible embodiments may involvedifferent session scoring schemas, measurement types and entry criteriafor use in the procedure of FIG. 2. It will now be described in moredetail how this may be done.

Each session scoring schema may have a priority and an entry condition,which is thus the above-mentioned “entry criteria”, which should befulfilled when selecting the session scoring schema for use in thisprocedure. If the entry criteria is met, this calculation schema can beused to obtain a total session-specific quality score Q. One option isthat the session scoring schema that has the highest priority isselected while the entry criteria is met.

There is also a last fallback schema at lowest priority that is alwaysthe same and applied if all other configured schemas could not beapplied due to their entry criteria. It will simply provide a sessionscore as an average of all individual session-specific Q scores.

It is an advantage that the solution and its embodiments describedherein may provide higher accuracy in the generation of session-specificquality scores and possibly also subjective perception scores. It maythus be possible to generate such scores for any type of service orexperience even if no dedicated MOS are available.

The Session Based Scoring module 108 may be introduced as apre-processing operation prior to a perception scoring operation. Thisway the main scoring will not get individual network measurements asinput, but quality scores determined per experienced service session.This session based scoring may be configured by definition of thescoring models 112 for each individual KPI, i.e. network measurement,and the Session Scoring Schemas 114.

Some further examples of a procedure for determining a session-specificquality score will now be described again with reference to FIG. 1. Inthis procedure, individual quality scores may be calculated fordifferent network measurements, such as KPIs, as shown in FIG. 3. Inthis example, the term KPI is used to represent any network measurementsalthough the example could be applicable for any type of networkmeasurements. A set of KPIs is taken as input and an individual scoringis applied to generate a quality score Q for each of the KPIs. In thisexample, a scoring function Q(v) is used that directly provides aquality score Q from the KPI value. Thus, Q1=4 is obtained from the KPIvalue MOS=2, Q2=10 is obtained from the KPI value A-KPI=1.0, and Q3=4 isobtained from the KPI value KPI 1=19. These scoring functions areconfigured for example in the database of Individual KPI scoring models112 shown in FIG. 1.

If the KPI is already a MOS originating from some previous scoringoperation, the KPI can be used directly. The scoring function would thenbe a linear function that only might apply a value range conversion.

Different predefined session scoring schemas may have differentpriorities. They may be stored in a data storage 114 for Session ScoringSchemas as shown in FIG. 1. For example, for different service orexperience types there could be different sets of session scoringschemas.

FIG. 4 shows four different examples of session scoring schemas denotedPriority n, Priority 2, Priority 1 and Priority 0, where Priority n hasthe highest priority. Each session scoring schema is defined by a set ofKPIs and requirements therefor. An entry criteria is specified bymarking as “Mandatory”, or “Mand”. The criteria is fulfilled if allMandatory KPIs are available in the session data, i.e. the receivednetwork measurements.

A weight is used in the calculation of the total session-specificquality score, as mentioned above which will also be described in moredetail later below, which basically indicates the weight this KPI shallhave in the total session-specific quality score.

The session scoring schema with priority 0 would always be applicable ifnone of the other schemas has entry criteria that are fulfilled. In thiscase the total session-specific quality score may be calculated as anaverage over all Q values obtained from the available KPIs since theKPIs in this session scoring schema have equal weight according to thesession scoring schema with priority 0 shown in FIG. 4.

FIG. 5 shows how the total session-specific quality score Q may becalculated based on the individual quality scores Q1-Q4 and therespective weights W1-W4 in the selected session scoring schema which isthe session scoring schema denoted Priority n in this example which issomewhat different from the session scoring schema Priority n shown inFIG. 4 in that both A-KPI and R-KPI are optional instead of mandatory.

The total session-specific quality score Q, here denoted Qsession, maybe calculated as a weighted average over the Q values obtained from theKPIs that are marked as either mandatory or optional. The weights in theschema are used as weights in the average.

KPIs that are not marked mandatory or optional in the requirement entrycan simply be ignored.

The value denoted E is the sum of all weights. E is 1 if all KPIs werepresent, i.e. have been received. This value is handed over to thefollowing perception scoring as initial weight for that scoring andtrust value of the session quality score.

FIG. 6 shows how the session-specific quality score Q may be calculatedwhen some KPI is missing, in this case R-KPI that would otherwisegenerate Q3=5 which is deleted in the figure.

If the missing KPI is mandatory, the next calculation schema shall beused that does not require that KPI.

If the missing KPI is optional, as in this example where R-KPI is markedOptional in the selected session scoring schema Priority n, as shown inFIGS. 5 and 6, the calculation of the score may be done as describedabove. The difference is that in this case one of the Q values ismissing from the weighted average. This situation is shown in FIG. 6where R-KPI is missing which means that Q3 cannot be obtained. In theselected session scoring schema Priority n, R-KPI is an optional KPI andthe session-specific quality score is only calculated from the remainingKPIs that were marked mandatory or optional.

A difference is then in the E value. It misses the weight from themissing KPI. The trust score of the overall session score is then only0.8 instead of 1. This way the subsequent perception scoring can treatthis particular session score with lower weight in order to accommodatethe lowered trust.

The solution and its embodiments described herein basically introduces ageneric and configurable scoring operation that may always provide asession-specific quality score.

The better the input data, the better will the resulting totalsession-specific quality score and perception score be, but also withless than optimal input data a session-specific quality score can beobtained.

If input data, i.e. network measurements, is missing, the weight of thetotal session-specific quality score of this service session will bereduced in subsequent perception scoring. This reflects that less trustis put in the total session-specific quality score.

Some example scenarios where the above-described solution andembodiments could be employed in practice, will now be outlined. It istypically of interest for a service provider to increase thesatisfaction of users when services are delivered by means of atelecommunication network. The embodiments described herein may help theservice provider in this respect by enabling a more accurate serviceevaluation by means of the session-specific quality score, which may beused according to the following examples 1-3.

1) The total session-specific quality score may be used to trigger analarm in network operation centers indicating that user experience isbad or degrading. Improvements that can be achieved by embodimentsherein may include that such alarms may be triggered at different timesand more accurately depending on the situation. “False” and misleadingalarms may also be avoided. As a result, maintenance technicians andresources may be used more efficiently.

2) Another use of the total session-specific quality score may be todisplay it on some information systems of a customer care agent or thelike. In this scenario the score may be displayed automatically aspersonal profile data whenever a customer is calling. In this way, thenumeric value of the displayed score may represent the customer'ssatisfaction more accurately.

3) The total session-specific quality score may also be used in more orless automatic proactive marketing as follows. In this case the scoremay be used as part of a decision logic used to decide if a marketingoperation shall be triggered and which customers shall be targeted bythe marketing operation. This marketing operation may e.g. involve anoffer of a promotion. Communication with the customer in this operationmay for example be made by an email or an SMS, being automaticallygenerated. The embodiments herein may thus provide a more accuratequality score that is better reflecting the individual perception of auser. This means that any marketing rules created based on the totalsession-specific quality score may more accurately select suitablecustomers for the resulting marketing operation.

The latter example 3 avoids for example that promotions are sent out tocustomers which are not likely to use the service anyway. Overall, thismay thus result in improved utilization of budget and resources.

The block diagram in FIG. 7 illustrates another detailed butnon-limiting example of how a score management node 700, which could bethe above-described score management node 100 in FIG. 1, may bestructured to bring about the above-described solution and embodimentsthereof. In this figure, the score management node 700 may thus beconfigured to operate according to any of the examples and embodimentsof employing the solution as described above, where appropriate, and asfollows. The score management node 700 in this example is shown in aconfiguration that comprises a processor “Pr”, a memory “M” and acommunication circuit “C” with suitable equipment for receiving andtransmitting data and messages in the manner described herein.

The communication circuit C in the score management node 700 thuscomprises equipment configured for communication with atelecommunication network, not shown, using one or more suitablecommunication protocols depending on implementation. As in the examplesdiscussed above, the score management node 700 is configured or arrangedto perform e.g. the actions of the flow chart illustrated in FIG. 2 inthe manner described above. These actions may be performed by means offunctional modules in the processor Pr in the score management node 700as follows.

The score management node 700 is arranged to support service evaluationof a service delivered by means of a telecommunication network. Thescore management node 700 thus comprises the processor Pr and the memoryM, said memory comprising instructions executable by said processor,whereby the score management node 700 is operable as follows.

The score management node 700 is configured to receive networkmeasurements related to at least one service event when the service isdelivered to the user. This receiving operation may be performed by areceiving module 700 a in the score management node 700, e.g. in themanner described for action 200 above. The score management node 700 isalso configured to calculate a quality score Q for each received networkmeasurement by applying a predefined scoring algorithm Q(v) on thenetwork measurement. This calculating operation may be performed by acalculating module 700 b in the score management node 700, e.g. in themanner described for action 202 above.

The score management node 700 is also configured to identify, among thecalculated quality scores, quality scores Q which are related to aspecific service session experienced by the user. This identifyingoperation may be performed by an identifying module 700 c in the scoremanagement node 700, e.g. in the manner described for action 204 above.The score management node 700 is also configured to determine a totalsession-specific quality score for the specific service session based onthe identified quality scores Q, wherein the total session-specificquality score is used for evaluation of the delivered service. Thisdetermining operation may be performed by a determining module 700 d inthe score management node 700, e.g. in the manner described for action206 above.

It should be noted that FIG. 7 illustrates some possible functionalunits in the score management node 700 and the skilled person is able toimplement these functional units in practice using suitable software andhardware. Thus, the solution is generally not limited to the shownstructure of the score management node 700, and the functional modules700 a-d may be configured to operate according to any of the featuresdescribed in this disclosure, where appropriate.

The embodiments and features described herein may thus be implemented ina computer program storage product comprising instructions which, whenexecuted on at least one processor, cause the at least one processor tocarry out the above actions e.g. as described for any of FIGS. 1-6. Someexamples of how the computer program storage product can be realized inpractice are outlined below, and with further reference to FIG. 7.

The processor Pr may comprise a single Central Processing Unit (CPU), orcould comprise two or more processing units. For example, the processorPr may include a general purpose microprocessor, an instruction setprocessor and/or related chips sets and/or a special purposemicroprocessor such as an Application Specific Integrated Circuit(ASIC). The processor Pr may also comprise a storage for cachingpurposes.

The memory M may comprise the above-mentioned computer readable storagemedium or carrier on which the computer program is stored e.g. in theform of computer program modules or the like. For example, the memory Mmay be a flash memory, a Random-Access Memory (RAM), a Read-Only Memory(ROM) or an Electrically Erasable Programmable ROM (EEPROM). The programmodules could in alternative embodiments be distributed on differentcomputer program products in the form of memories within the scoremanagement node 700.

It was mentioned above that a perception score P may be determined basedon one or more total session-specific quality scores, the perceptionscore P reflecting a user experience of a service. For example, theperception score P can be used in the service evaluation as anestimation of the users' opinion and it is possible to obtain Pautomatically after every time a service is delivered to a user.Further, since the perception score P is basically calculated fromtechnical measurements in the network related to a specific servicesession, it is possible to evaluate the performance during that servicesession based on the perception score P. Since the calculated P is thus“session-specific”, any service session that performs less thansatisfactorily can be identified and analyzed to improve the servicedelivery.

The score management node 100 may thus determine the perception score Pbased on the total session-specific quality score, as of action 208.Some examples of how this could be done will now be described in moredetail, which refer to “network measurements” in general whichcorrespond to the above-described network measurements v received inaction 200.

An example of a procedure will thus now be described for how theperception score P may be determined by the score management node basedon network measurements which could thus be the above-mentionedsession-specific network measurements.

The perception score P may be determined by the score management node asfollows. The received network measurements v can be seen as “raw data”being used as input in this procedure. In this example, a quality scoreQ reflecting the user's perception of quality of a delivered service andan associated significance S reflecting the user's perception ofimportance of the delivered service, are determined based on the networkmeasurements, where Q corresponds to the above-describedsession-specific quality score which will be denoted “quality score” forshort hereafter. In this operation, Q and S may be determined byapplying predefined functions on the network measurements, which will beexplained in more detail later below. The perception score P is thenderived from the quality score Q which is weighted by its associatedsignificance S. Basically, the greater significance S the greaterinfluence has the associated quality score Q on the resulting perceptionscore P.

Before calculating the perception score P, the quality score Q andassociated significance S may also be modified in this procedure basedon a set of predefined influence factors valid for the user and thedelivered service. These influence factors may be related to userexpectation considering various characteristics of the user, correlationof different service events occurring within a certain time frame, andfading memory of the user which reduces the significance S of a serviceevent over time. The perception score P is then calculated from themodified quality score Q and associated significance S, and theresulting perception score P can then be made available for supportingevaluation of the service. By using this procedure, the perception scoreP can be seen as a model for how the user is expected to perceive theservice given the circumstances of the delivered service, which model isbased on objective and technical network measurements.

Next, the operation of modifying Q and S according to the aboveinfluence factors is performed. In this way, Q and S are determinedpurely from the raw data, i.e. the received network measurements, whileQ and S are adjusted by considering the circumstances of the serviceevent which produce the above influence factors, thereby making Q and Smore adapted to the actual situation of the delivered service.

Further, the operation of calculating the perception score P from themodified Qm weighted by its associated and modified Sm is performed.Having generated the resulting perception score P, the score managementnode makes P available for evaluation of the service, e.g. by saving itin a suitable storage or sending it to a service evaluation system orcenter. For example, P may be sent to the service evaluation system orstorage in an http message or an ftp message over an IP network. Theservice evaluation system or storage may comprise an SQL (StructuredQuery Language) database or any other suitable type of database.

The quality score Q and associated significance S are thus modifiedgradually in multiple steps such that the output of modified Q′ and/orS′ is used as input for further modification, until the thus processeddata is used for calculation of P.

There are several advantages of this procedure as compared toconventional ways of obtaining a user's opinion about a service. First,the perception score P is a quite accurate estimation of the users'opinion of the service event considering the prevailing circumstances,and it is possible to obtain P automatically and continuously inreal-time, basically after every time a service is delivered to a user.There are thus no restrictions regarding the number of users nor theextension of time which makes it possible to obtain a quiterepresentative perception score P. Second, the perception score P iscalculated from technical measurements made in the network related tothe service usage which are truthful and “objective” as such, also beingreadily available, thereby avoiding any dependency on the user's memoryand willingness to answer a survey or the like. Third, it is notnecessary to spend time and efforts to distribute surveys and to collectand evaluate responses, which may require at least a certain amount ofmanual work.

Fourth, it is possible to gain further knowledge about the service bydetermining the perception score P selectively, e.g. for specific typesof services, specific types of network measurements, specific users orcategories of users, and so forth. Fifth, it is also possible to trace atechnical issue that may have caused a “bad” experience of a deliveredservice by identifying which measurement(s) have generated a lowperception score P. It can thus be determined when and how a service wasdelivered to a presumably dissatisfied user, as indicated by theperception score P, and therefore a likely technical shortcoming thathas caused the user's dissatisfaction can also be more easilyidentified. Once found, the technical issue can be eliminated orrepaired. Different needs for improvement of services can also beprioritized based on the knowledge obtained by the perception score P.

It was mentioned above that Q and S may be determined by applyingpredefined functions or scoring algorithms on the network measurements.Particularly in the above action 202, the score management node 100calculates a quality score Q for each received network measurement byapplying a predefined scoring algorithm Q(v) on the network measurementsreceived in action 200. For example, Q may be determined by applying afirst function Q(v) on the network measurements v, and S may bedetermined by applying a second function S(v) on the networkmeasurements v. Further, the first and second predefined functions Q(v)and S(v) may be dependent on a type of the network measurements used asinput to the functions so that a function applied on, say, measurementof data rate is different from a function applied on measurement of calldrop rate, to mention two non-limiting but illustrative examples.

The score management node may then modify the determined quality score Qand associated significance S of each service event based on apredefined influence factor applied in each intermediate scoring module.This means that Q and S, or at least one of Q and S, may be modifiedbased on a first predefined influence factor. The once modified Q′ andS′ may then be modified further based on a second predefined influencefactor. The twice modified Q″ and S″ may then be modified further basedon a third predefined influence factor, and so forth. Any number of suchinfluence factors may be used.

The predefined influence factors may comprise at least two of:

A) User expectation. In this example, a user profile withcharacteristics pertaining to the user is defined and at least one usergroup that matches the user profile is identified. The quality score Qand associated significance S can then be modified based on predefinedgroup-specific parameters valid for the at least one identified usergroup. The group-specific parameters have thus been defined for a usergroup to basically describe the user group. Thus, the user can therebybe described by means of membership in one or more of these user groupsdepending on how relevant the group-specific parameters are to the user.

B) Correlation of different service events. In this example, thesignificance S of a quality score Q for a first service event ismodified by multiplying a correlation factor F reflecting a correlationbetween the first service event and a second service event when thefirst and second service events have both occurred within a certain timeframe. For example, the correlation factor F may be greater the closertwo service events are in time assuming that if one of the events hasparticularly high significance to the user the other event will also belikely to have high significance to the user if the two service eventsoccur within a short time frame.

C) Fading memory of the user. In this example, the significance S ofeach quality score Q is reduced over time according to a predefinedSignificance Reduction Rate, SRR assuming that a user's memory of aservice event tends to fade over time and this can be compensated byreducing the significance of the service event over time accordingly. Byreducing the significance S over time to simulate the user's fadingmemory of the service event, the perception score P will likewise bereduced over time. The SRR may be defined to form a step-like functionwhich reduces S in distinct steps over time until it finally reacheszero assuming that the service event is virtually forgotten by the userat this point.

In this way, Q and S have been modified according to the predefinedinfluence factors as exemplified above and the resulting modifiedquality score “Qm” and associated significance “Sm” are used as input inthe next action where the score management node calculates theperception score P based on the modified quality score Qm and associatedmodified significance Sm. Finally, the calculated perception score P maybe made available for use in the service evaluation, e.g. by sending Pto a suitable service evaluation system or storage. The protocol used inthis communication may be e.g. the hyper-text transfer protocol http orthe file transfer protocol ftp, and the perception score P may be sentto the service evaluation system or storage in an http message or an ftpmessage over an IP network. The service evaluation system or storage maycomprise an SQL (Structured Query Language) database or any othersuitable type of database.

The perception score P may be calculated according to different possibleprocedures as follows. In one example, the score management node maycalculate the perception score P for multiple service events of servicedelivery to the user as an average of modified quality scores Qm for theservice events weighted by their associated modified significances Sm.In this case, the score management node may calculate the perceptionscore P_(N) for N service events of service delivery to the useraccording to the following formula:

$P_{N} = \frac{\sum\limits_{n = 1}^{N}{Q_{s}S_{n}}}{\sum\limits_{n = 1}^{N}S_{n}}$

where Q_(a) is the modified quality score for a service event n andS_(n) is the associated modified significance for said service event n.In other words, the sum of all N quality scores weighted by theirsignificances is divided by the sum of all the N significances.

The network measurements may be made during a predefined time interval.Further, the score management node may update the perception score Pafter a new service event n based on a previous perception score P_(n-1)calculated for a previous time interval or service event and a qualityscore Q_(n) and associated significance S_(n) determined for the newservice event n, according to the following formula:

$P_{n} = \frac{{P_{n - 1}S_{{sum},{n - 1}}} + {Q_{n}S_{n}}}{S_{{sum},{n - 1}} + S_{n}}$

whereS_(sum,n)=S_(sum,n-1)+S_(n) and P_(n) is the updated perception score.In this way, the perception score

P can be kept up-to-date after each new service event by using the abovesimple calculation which adds the influence of the new service event non the total P.

In further examples, the score management node may identify at least onetype of service for which a modified significance S satisfies athreshold condition. If so, the score management node may then providethe identified at least one type of service as input to root causeanalysis when the perception score P is changed significantly. The term“root cause analysis” refers to a procedure for tracing a technicalreason for why a service has e.g. been delivered poorly, which procedureas such is somewhat outside the scope of this disclosure. In thisembodiment the root cause analysis is deemed to be warranted if theperception score P has changed significantly, particularly when P hasdecreased which indicates that the user is expected to be dissatisfiedwith the service as shown by the network measurement(s).

The threshold condition is thus used for finding service events ofunexpected perception score P, either surprisingly low or high. Thisalso makes it easy to exactly identify individual service events thatmay have caused a “bad” experience of a delivered service. For example,the threshold condition may require that the modified significance S ishigh which indicates that the corresponding service event has had agreat influence on the changed P. Thereby, the search for a technicalreason can be focused on that service event to some extent.

It was mentioned above that the score management node may identify atleast one type of service for which a modified significance S satisfiesa threshold condition, and that the identified at least one type ofservice may then be provided as input to root cause analysis when theperception score P is changed significantly. Examples of how this can bedone will now be described. It is assumed that the resulting modifiedsignificance S can be detected and collected, e.g. the output from thelast intermediate scoring module being the modified significance Sm, inorder to generate a table with services that have generated the highestsignificances as follows.

The final modified significance S of a single service event may thus beused in order to determine what type of service did get the highestoverall significance. In this case the significances determined for acertain service type are summed up and the sum value is stored. In thisway, a significance table can be built that shows which types ofservices did have the highest significance in the calculation of theperception score. The significance table can be sorted according to thesignificance sums resulting in a list with the most significant serviceevent on top of the list. This shows what type of service has producedthe highest weight in the calculation of the perception score P.

An example of such a significance table comprises entries for differentservice types and their resulting significance sum, the number ofscorings of service events and a calculated average of the significancefor all service events. Whenever a new scoring for a service type Txwith a significance S is obtained, S is added to the significance sumS_Tx of the service type Tx. In this table, also the number of scoringsand the average significance are kept for each service type. Thisprovides further information indicating whether the significance of aservice type is coming from a small number of very significant serviceevents or from a large number of less significant ones. This may providefurther insights into the service event history of the user and the rootcause for the perception score.

A table like this is associated with the perception score P. Thus forevery perception score P, a table of the most significant experienceevents can be made available. As similar to the perception score P, thistable is user specific and this kind of table can be generated for eachuser.

It may be of interest to find out why the perception score P hasincreased or declined, and this significance table can indicate whattypes of services had the greatest influence on changes in theperception score. Further investigations in the root cause analysis canthen focus on these service types accordingly.

Another example of a procedure will now be described for how theperception score P may be determined by the score management node basedon network measurements which could thus be the above-mentionedsession-specific network measurements.

In this example, a quality score Q reflecting the user's perception ofquality of a delivered service, is determined by applying a firstfunction Q(v) on the network measurements v. Further, an associatedsignificance S reflecting the user's perception of importance of thedelivered service, is also determined by applying a second function S(v)on the network measurements v. The quality score Q and its associatedsignificance S may be determined in this manner for each networkmeasurement by the score management node. The above-mentioned first andsecond functions Q(v), S(v) may be predefined for a particularmeasurement type and they may be maintained in the score managementnode. Different variants of the first and second functions Q(v), S(v)may thus be maintained for different measurement types which will bedescribed in more detail later below.

The perception score P of the received network measurements v is thenderived from the quality scores Q which are weighted by their associatedsignificances S. Basically, the greater significance S the greaterinfluence has the associated quality score Q on the resulting perceptionscore P. This example is directed to describe how the above qualityscore Q, significance S and perception score P can be determined.

Before calculating the perception score P, one or both of the qualityscore Q and associated significance S may be modified in this proceduredepending on whether the quality score Q determined for a new servicedelivery event deviates significantly from a “normal”, i.e. expected,level of the perception score P calculated previously. For example, theuser may be assumed to expect basically the same level of quality “asusual” whenever a service is delivered. If the quality, as determinedfrom one or more network measurements of a new service delivery event,suddenly departs from the expected level, the user can further beassumed to be “surprised” by the unexpected quality level and e.g. thesignificance S of that event may therefore be increased.

The score management node may further operate to modify the qualityscore Q and its associated significance S in order to compensate forvarious circumstances at the respective service delivery, e.g. includingthe user's expectations of the service delivery as mentioned above. Theuser's expectations are basically indicated by a previously determinedoverall perception score valid for one or more previous servicedeliveries. For example, one or both of the quality score Q and theassociated significance S may be modified assuming that Q and/or S of anew service event may be impacted depending on a deviation between thenew quality score Q and a previous perception score P, which deviationeffectively reflects a degree of assumed “surprise” to the user.

Having generated the resulting perception score P, the score managementnode makes P available for evaluation of the service, e.g. by saving itin a suitable storage or sending it to a service evaluation system orcenter. By using this procedure, the perception score P can be seen as amodel for how the user is expected to perceive the service given thecircumstances of the delivered service, which model is based onobjective network measurements. Thus, P is a quantification of theuser's assumed perception of the service deliveries.

There are several advantages of this procedure as compared toconventional ways of obtaining a user's opinion about a service. First,the perception score P is a quite accurate estimation of the users'opinion of the service event considering the prevailing circumstances,and it is possible to obtain P automatically and continuously inreal-time, basically after every time a service is delivered to a user.There are thus no restrictions regarding the number of users nor theextension of time which makes it possible to obtain a quiterepresentative perception score P. Second, the perception score P iscalculated from technical measurements in the network related to theservice usage which are true and “objective” as such, also being readilyavailable, thereby avoiding any dependency on the user's memory andwillingness to answer a survey or the like. Third, it is not necessaryto spend time and efforts to distribute surveys and to collect andevaluate responses, which may require at least a certain amount ofmanual work.

Fourth, it is possible to gain further knowledge about the service bydetermining the perception score P selectively, e.g. for specific typesof services, specific types of network measurements, specific users orcategories of users, and so forth. Fifth, it is also possible to trace atechnical issue that may have caused a “bad” experience of a deliveredservice by identifying which measurement(s) have generated a lowperception score P. It can thus be determined when and how a service wasdelivered to a presumably dissatisfied user, as indicated by theperception score P, and therefore a likely technical shortcoming thathas caused the user's dissatisfaction can also be more easilyidentified. Once found, the technical issue can be eliminated orrepaired. Different needs for improvement of services can also beprioritized based on the knowledge obtained by the perception score P.Further features and advantages will be evident in the description ofexamples that follows.

It was mentioned above that different variants of the first and secondfunctions may thus have been predefined for different networkmeasurement types, e.g. being maintained in the score management node.For example, a variant of function Q(v) or S(v) applied on, say, ameasurement of data rate is different from a variant of function Q(v) orS(v) applied on a measurement of call drop rate, to mention anon-limiting but illustrative example.

In another example, the score management node may maintain associationsbetween different network measurement types and different variants ofthe first and second functions, e.g. in a suitable document or datastorage. In this example, the score management node may select a variantof the first and second functions according to said associations fordetermining the quality score Q and associated significance S for eachnetwork measurement. When receiving a network measurement, the scoremanagement node is thus able to identify the type of the networkmeasurement and select a variant of the first and second functionsaccording to the identified measurement type. In further examples, eachof the first and second functions may be a discrete function or acontinuous function.

In a possible example, the score management node may determine multiplepairs of the quality score Q and associated significance S based on thenetwork measurements, e.g. one pair for each network measurement. A pairof Q and S is thus determined for each service event based on thenetwork measurement for that service event. The score management nodemay then calculate the perception score P as an average of the qualityscores Q weighted by their associated significances S in all the abovepairs of Q and S. In a further example, this may be done such that whenthe number of service events is N, the score management node calculatesthe perception score P_(N) for the N events of service delivery to theuser as

$P_{N} = \frac{\sum\limits_{n = 1}^{N}{Q_{s}S_{n}}}{\sum\limits_{n = 1}^{N}S_{n}}$

where Q_(n) is the quality score determined for each service event n andS_(n) is the associated significance determined for said service eventn. In other words, the sum of all N quality scores weighted by theirsignificances is divided by the sum of all the N significances. Thereby,the quality score Q_(n) for each service event n will impact the overallperception score P_(N) according to its associated significance S_(n)and P_(N) will thus become an accurate representation of the user'sperception of quality of service delivery across all service events N.These examples may have the advantage that a perception score can beobtained that reflects the user's experience of a service over aspecific selection of service events N. The overall perception scoreP_(N) may thus be calculated for any selection of service events N asdesired.

Alternatively, an “accumulated” perception score P may be obtained andupdated after each new service event as follows. Thus in anotherexample, the score management node may update the perception score Pafter a new service event n based on a previous perception score P_(n−1)calculated for a previous time interval or service event and a qualityscore Q_(n) and associated significance S_(n) determined for the newservice event n, as

$P_{n} = \frac{{P_{n - 1}S_{{sum},{n - 1}}} + {Q_{n}S_{n}}}{S_{{sum},{n - 1}} + S_{n}}$

where

$S_{{sum},n} = {\sum\limits_{n = 1}^{N}S_{n}}$

and P_(n) is the updated perception score. In this way, the perceptionscore P can be kept up-to-date after each new service event by using theabove simple calculation which adds the influence of the new serviceevent n on the total P. This example may have the advantage that theupdated perception score P_(n) reflects the user's experience of aservice in a “continuous” manner by always taking the latest serviceevent into account.

In yet another example, the score management node may determine theperception score P for a service of a particular type by calculating theperception score P according to the above procedure for multiple usersupon service delivery to the users with a service of said particulartype. The additional information provided by this example may be used tosupport or facilitate tracing of any technical issue that may cause alow perception score P for the particular service type.

It was mentioned above that the score management node may maintainassociations between the respective network measurement types and thevariants of the first and second functions Q(v), S(v). Such variants ofthe functions may be associated with network measurement types in atable where a variant Q1(v) of the first function and a variant S1(v) ofthe second function are associated with a measurement “type 1”. Further,another variant Q2(v) of the first function and another variant S2(v) ofthe second function are associated with another measurement “type 2”,and so forth. By identifying the measurement type of an incoming networkmeasurement, the score management node can thus find the correctvariants of the first and second functions Q(v), S(v) in this table andapply them accordingly to determine Q and S.

Another table may comprise variants of the functions Q(v) and S(v) fortwo network measurement types, video-frame rate and the time needed todownload a web page. It was further mentioned above that either of thefirst and second functions may be a discrete function or a continuousfunction. Thus, each of the first function Q(v) and the second functionS(v) may be a discrete function for the measurement type video-framerate, such that Q increases and S decreases in discrete steps uponincreased video-frame rate v. Q may increase in discrete steps uponincreased video-frame rate v in frames per second, fps. For example, Q=0when v is lower than 10, Q=1 when v is between 10 and 15, Q=2 when v isbetween 15 and 20, Q=3 when v is between 20 and 25, and Q=4 when v ishigher than 25. On the other hand, each of the first function Q(v) andthe second function S(v) may be a continuous function for themeasurement type time needed to download a web page, meaning that Qdecreases and S increases continuously upon increased time needed todownload a web page.

It should be noted that the functions Q(v) and S(v) for the measurementtype video-frame rate produce higher Q and lower S values the higher thevideo-frame rate is, while the functions Q(v) and S(v) for themeasurement type video-frame rate produce lower Q and higher S valuesthe longer time needed to download a web page. By these variants offunctions Q(v) and S(v), it is assumed that Q is relatively low and S isrelatively high when the network measurement v indicates “bad” quality,either by low video-frame rate or by higher the time needed to downloada web page, and vice versa.

While the solution has been described with reference to specificexemplifying embodiments, the description is generally only intended toillustrate the inventive concept and should not be taken as limiting thescope of the solution. For example, the terms “score management node”,“network measurement”, “perception score”, “service event”, “qualityscore”, “service session”, “session scoring schema” and “entry criteria”have been used throughout this disclosure, although any othercorresponding entities, functions, and/or parameters could also be usedhaving the features and characteristics described here.

1. A method performed by a score management node for supporting serviceevaluation of a service delivered to a user by means of atelecommunication network, the method comprising: receiving networkmeasurements (v) related to at least one service event when the serviceis delivered to the user, calculating a quality score Q for eachreceived network measurement by applying a predefined scoring algorithmQ(v) on the network measurement, identifying , among the calculatedquality scores, quality scores Q which are related to a specific servicesession experienced by the user, and determining a totalsession-specific quality score for the specific service session based onthe identified quality scores Q, wherein the total session-specificquality score is used for evaluation of the delivered service.
 2. Themethod of claim 1, wherein the total session-specific quality score isdetermined based on a session scoring schema comprising a set ofweighted measurement types such that the total session-specific qualityscore is calculated as a weighted average of the identified qualityscores Q where each quality score Q is weighted according to the type ofmeasurement used for determining said quality score Q.
 3. The method ofclaim 2, wherein the session scoring schema is selected from a set ofpredefined session scoring schemas depending on which types of networkmeasurements have been made for the at least one service event.
 4. Themethod of claim 3, wherein each session scoring schema in the set ofpredefined session scoring schemas comprises an entry criteriaindicating at least one mandatory measurement type and/or at least oneoptional measurement type.
 5. The method of claim 3, wherein the sessionscoring schema is selected further depending on priorities assigned tothe session scoring schemas in the set of predefined session scoringschemas.
 6. The method of claim 3, wherein the session scoring schemasin the set of predefined session scoring schemas are defined fordifferent services or experience types.
 7. The method of claim 2,wherein the measurement types include one or more Key PerformanceIndicators, KPIs.
 8. The method of claim 1, further comprisingdetermining a perception score P based on the total session-specificquality score, the perception score P reflecting a user experience of aservice.
 9. The method of claim 8, wherein the perception score P isdetermined based on total session-specific quality scores calculated formultiple service sessions.
 10. The method of claim 1, wherein thenetwork measurements are related to any of: the time needed to downloaddata, the time from service request until delivery, call drop rate, datarate, and data error rate.
 11. The method of claim 1, wherein thenetwork measurements are received in a message according to thehyper-text transfer protocol http or the file transfer protocol ftp. 12.A score management node arranged to support service evaluation of aservice delivered to a user by means of a telecommunication network, thescore management node comprising a processor and a memory containinginstructions executable by the processor, whereby the score managementnode is configured to: receive network measurements related to at leastone service event when the service is delivered to the user, calculate aquality score Q for each received network measurement by applying apredefined scoring algorithm Q(v) on the network measurement, identify,among the calculated quality scores, quality scores Q which are relatedto a specific service session experienced by the user, and determine atotal session-specific quality score for the specific service sessionbased on the identified quality scores Q, wherein the totalsession-specific quality score is used for evaluation of the deliveredservice.
 13. The score management node of claim 12, wherein the scoremanagement node is configured to determine the total session-specificquality score based on a session scoring schema comprising a set ofweighted measurement types such that the total session-specific qualityscore is calculated as a weighted average of the identified qualityscores Q where each quality score Q is weighted according to the type ofmeasurement used for determining said quality score Q.
 14. The scoremanagement node of claim 13, wherein the score management node isconfigured to select the session scoring schema from a set of predefinedsession scoring schemas depending on which types of network measurementshave been made for the at least one service event.
 15. The scoremanagement node of claim 14, wherein each session scoring schema in theset of predefined session scoring schemas comprises an entry criteriaindicating at least one mandatory measurement type and/or at least oneoptional measurement type.
 16. The score management node of claim 14,wherein the score management node is configured to select the sessionscoring schema further depending on priorities assigned to the sessionscoring schemas in the set of predefined session scoring schemas. 17.The score management node of claim 14, wherein the session scoringschemas in the set of predefined session scoring schemas are defined fordifferent services or experience types.
 18. The score management node ofclaim 13, wherein the measurement types include one or more KeyPerformance Indicators, KPIs.
 19. The score management node of claim 12,wherein the score management node is configured to determine aperception score P based on the total session-specific quality score,the perception score P reflecting a user experience of a service. 20.The score management node of claim 19, wherein the score management nodeis configured to determine the perception score P based on totalsession-specific quality scores calculated for multiple servicesessions.
 21. The score management node of claim 12, wherein the networkmeasurements are related to any of: the time needed to download data,the time from service request until delivery, call drop rate, data rate,and data error rate.
 22. The score management node of claim 12, whereinthe score management node is configured to receive the networkmeasurements in a message according to the hyper-text transfer protocolhttp or the file transfer protocol ftp.
 23. A computer program productcomprising a non-transitory computer readable medium storinginstructions which, when executed on at least one processor, cause theat least one processor to carry out the method according to claim 1.